fix: Lambda GPU image security hardening#5733
Merged
Conversation
The following CVEs are now fixed in AL2023 and will be resolved by the existing dnf upgrade --security in runtime stages: - CVE-2022-46908 (sqlite, ALAS2023-2025-971) - CVE-2024-56171 (libxml2, ALAS2023-2025-896) - CVE-2025-24928 (libxml2, ALAS2023-2025-896) - CVE-2025-13151 (libtasn1, ALAS2023-2026-1395) - CVE-2025-13601 (glib2, ALAS2023-2025-1311) - CVE-2025-1390 (libcap, ALAS2023-2025-897) - CVE-2025-14087 (glib2, ALAS2023-2025-1349) - CVE-2025-15467 (openssl, ALAS2023-2026-1406) - CVE-2025-15468 (openssl, ALAS2023-2026-1434) - CVE-2025-4802 (glibc, ALAS2023-2025-1001) Reduces allowlist from 35 to 25 entries. Signed-off-by: Junpu Fan <junpu@amazon.com>
Signed-off-by: Junpu Fan <junpu@amazon.com>
The workflow triggered but build-images was skipped because the check-changes build-change filter didn't include the allowlist path. Signed-off-by: Junpu Fan <junpu@amazon.com>
The NVIDIA CUDA base image pins to an older AL2023 release version, so dnf upgrade --security misses patches available in newer releases. Adding --releasever latest ensures all available security fixes are applied, including fixes for sqlite, libxml2, libtasn1, glib2, libcap, openssl, and glibc CVEs. Signed-off-by: Junpu Fan <junpu@amazon.com>
With --releasever latest, dnf upgrade --security now picks up fixes for libxml2, glib2, expat, libarchive, gnupg2, openssl, and libxslt CVEs. Reduces allowlist from 25 to 14 entries. Remaining entries are Python stdlib (tarfile), bundled setuptools/ urllib3, NVIDIA CUDA base image Go tooling, and unpublished CVEs that cannot be resolved via OS package updates. Signed-off-by: Junpu Fan <junpu@amazon.com>
Install awslambdaric via uv instead of inheriting it from the Lambda base image. Copy only Python binary/stdlib/lib from lambda-python stage, drop site-packages entirely, and install all deps (including awslambdaric==3.1.1 and pip==25.3) via uv so we own the full dependency tree. Remove pip after OSS compliance step runs. This eliminates the three pip-bundled CVEs that could not be patched via requirements pinning: - CVE-2024-6345 (setuptools RCE via pip/_vendor/pkg_resources) - CVE-2025-47273 (setuptools path traversal via pip/_vendor/pkg_resources) - CVE-2026-21441 (urllib3 decompression bomb via pip/_vendor/urllib3==1.26.20) Allowlist reduced from 14 to 11 entries. Signed-off-by: Junpu Fan <junpu@amazon.com>
pip is needed only by setup_oss_compliance.sh during the build. Install it via uv in each builder stage rather than pinning it in requirements files, so it never appears as an explicit dependency. Signed-off-by: Junpu Fan <junpu@amazon.com>
setup_oss_compliance.sh calls python -m pip at build time. Since we no longer inherit site-packages from the Lambda base image, pip must be explicitly installed via requirements. Signed-off-by: Junpu Fan <junpu@amazon.com>
3685312 to
a6889ff
Compare
ECR scan detects setuptools 38.5.0 inside pip/_vendor/pkg_resources bundled within ensurepip/_bundled/pip-25.3-py3-none-any.whl. The bundled wheel is only used to bootstrap pip into new virtualenvs, which never happens in a Lambda runtime. Remove it. Signed-off-by: Junpu Fan <junpu@amazon.com>
…positives ECR scanner attributes setuptools 38.5.0 to pip_licenses-5.5.1.dist-info/METADATA but no such version exists in the runtime image. setuptools==78.1.1 is installed and both pip and ensurepip/_bundled (which contained the old vendored copy) have been removed from the runtime image. Signed-off-by: Junpu Fan <junpu@amazon.com>
deep_learning_container.py imports botocore.session for IMDS calls. Previously provided by Lambda base image bundled site-packages; now that we install site-packages ourselves, botocore must be explicit. cupy and pytorch requirements already include boto3 (which pulls botocore). Signed-off-by: Junpu Fan <junpu@amazon.com>
Consistent with cupy and pytorch requirements which both pin boto3==1.40.4. Signed-off-by: Junpu Fan <junpu@amazon.com>
setuptools 82.0.1 vendors jaraco.context 6.1.0 which fixes CVE-2026-23949. Remove CVE-2026-23949 from allowlist. Signed-off-by: Junpu Fan <junpu@amazon.com>
…dates All 4 Go CVEs (CVE-2025-47912, CVE-2025-58188, CVE-2026-25679, CVE-2026-27142) are in aws-lambda-rie v1.33 built with Go 1.25.7. Fixed in Go 1.25.8 (2026-03-05) but no new RIE release yet. We use releases/latest so will auto-resolve. Extended review_by to 2026-06-12. Signed-off-by: Junpu Fan <junpu@amazon.com>
vulnerablePackages entries include filePath indicating exactly where the scanner found the vulnerable package. Include it in the error log to make false positives and vendored copies immediately diagnosable. Signed-off-by: Junpu Fan <junpu@amazon.com>
…s, remediation - Use top-level fixAvailable field to skip unfixable CVEs (more reliable than per-package fixedInVersion heuristic) - Add inspectorScore, exploitAvailable, epss.score to failure header - Add per-package remediation command (e.g. 'sudo dnf check-update') - Retain filePath from previous commit Signed-off-by: Junpu Fan <junpu@amazon.com>
Signed-off-by: Junpu Fan <junpu@amazon.com>
- CVE-2024-6345, CVE-2025-47273: ECR scanner false positives on pip_licenses dist-info - CVE-2026-25679, CVE-2026-27142: Go CVEs in aws-lambda-rie v1.33 (Go 1.25.7), fixed in Go 1.25.8, pending new RIE release All tarfile stdlib CVEs and GnuPG CVE-2026-24882 dropped from allowlist — fixAvailable=NO so the scan filter now correctly ignores them. Signed-off-by: Junpu Fan <junpu@amazon.com>
Remove pip deletion from all 3 runtime stages — customers building on top of these images need pip to install additional packages. Keep ensurepip/_bundled removal (bootstrap artifact, not needed at runtime). Update allowlist reason for CVE-2024-6345 and CVE-2025-47273: setuptools 38.5.0 is vendored inside pip/_vendor/pkg_resources but is never invoked at Lambda runtime; only the installed setuptools==82.0.1 is used. Signed-off-by: Junpu Fan <junpu@amazon.com>
Signed-off-by: Junpu Fan <junpu@amazon.com>
CVE-2024-6345 and CVE-2025-47273 are confirmed ECR scanner false positives — scanner attributes setuptools 38.5.0 to pip_licenses dist-info/METADATA regardless of whether pip is present or absent in the image. Signed-off-by: Junpu Fan <junpu@amazon.com>
Latest pip release. Note: pip still vendors urllib3==1.26.20 internally (pip/_vendor/urllib3) — this is a known upstream issue; pip has been stuck on urllib3 1.x for compatibility reasons regardless of version. Signed-off-by: Junpu Fan <junpu@amazon.com>
These CVEs are permanently unfixable as long as pip is present in the image (pip vendors old setuptools/urllib3 by design). No review date needed. Signed-off-by: Junpu Fan <junpu@amazon.com>
zhuofuAMZ
approved these changes
Mar 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Comprehensive security hardening for Lambda GPU runtime images, reducing the ECR scan allowlist from 35 → 4 entries.
Changes
OS patching
dnf upgrade -y --security --releasever latestto all 3 runtime stages to pick up AL2023 security fixes blocked by the NVIDIA base image's pinned release versionPython dependency hardening
COPY --from=lambda-python /var/lang /var/langwith selective copy ofbin,include,libonly, thenrm -rf site-packagesuvso we own the full dependency tree (no Lambda-bundled boto3/botocore/urllib3/etc.)awslambdaric==3.1.1,boto3==1.40.4,setuptools==82.0.1,pip==26.0.1,urllib3==2.6.3, etc.ensurepip/_bundled— CPython-bundled pip wheel containing old vendored setuptools/urllib3; not needed at runtimepipin runtime images for customer extensibility (consistent with existing Lambda runtime images)CVEs resolved (31 removed from allowlist)
dnf upgrade --releasever latestensurepip/_bundledfixAvailable=NO— scan filter correctly ignores unfixable CVEsfixAvailable=NO— scan filter ignoresaws-lambda-rie; corrected allowlist reasonRemaining allowlist (4 entries — irreducible)
pip/_vendor/pkg_resources— structural limitation of pip, present in all pip versions; vendored code is unreachable by user code and never invoked at Lambda runtimeaws-lambda-riev1.33 (Go 1.25.7); fixed in Go 1.25.8, pending new RIE release; usingreleases/latestso will auto-resolveECR scan tooling improvements
filePathin failure output (critical for diagnosing vendored copies and false positives)inspectorScore,exploitAvailable,epss.score,remediationto failure outputfixAvailablefield to skip genuinely unfixable CVEs (eliminates need to allowlist stdlib/OS CVEs with no available fix)